The emergence of movement units through learning with noisy efferent signals and delayed sensory feedback

Rapid human arm movements often have velocity profiles consisting of several bell-shaped acceleration-deceleration phases, sometimes overlapping in time and sometimes appearing separately. We show how such sub-movement sequences can emerge naturally as an optimal control policy is approximated by a reinforcement learning system in the face of uncertainty and feedback delay. The system learns to generate sequences of pulse-step commands, producing fast initial sub-movements followed by several slow corrective sub-movements that often begin before the initial sub-movement has completed. These results suggest how the nervous system might efficiently control a stochastic motor plant under uncertainty and feedback delay.